# Pre-training Optimization

MrT5 is an efficient byte-level language model based on ByT5 improvements, reducing input sequence length by approximately 50% through dynamic token merging technology

A special token adjusted version optimized based on Meta-Llama-3-70B, fixing fine-tuning issues caused by untrained special tokens in the original model

A medium-sized BERT language model using Masked Language Modeling (MLM) as the pre-training objective.

A medium-scale BERT language model that uses first character prediction as the pre-training objective.

A Chinese version of the PAGASUS-large model specialized in text summarization tasks, trained on the PEGASUS architecture with optimizations for Chinese tokenization.

This model is based on the tapas-base architecture, optimized for table input through pre-training, enhancing reasoning capabilities for QA tasks.